Drop dmesg, df and mount output, add build log scavenging and uploading#3984
Drop dmesg, df and mount output, add build log scavenging and uploading#3984
Conversation
This was introduced to see if it can help us pinpoint some weird issues we used to have when suddenly we were not able to create a directory or stuff like that. It was actually never useful, and is always annoying to scroll through this output to see the actual error that caused the build to fail. Signed-off-by: Krzesimir Nowak <knowak@microsoft.com>
This searches for portage logs and configure logs after a build, and uploads them to bincache. This is currently only done for SDK builds, SDK container builds and package builds. This probably could be extended to catch logs for sysext builds, but this was annoying to implement, IIRC. Signed-off-by: Krzesimir Nowak <knowak@microsoft.com>
This is to give ci-automation a chance to scavenge the logs that are stored in the tmp directory. Signed-off-by: Krzesimir Nowak <knowak@microsoft.com>
|
Looks good to me, wanted to get rid of that dmesg as well - the dmesg is inherited from the host and if the host has been online for ages, it creates an impossible to scroll - up - > to the actual error. |
Test report for 4685.0.0+nightly-20260429-2100 / amd64Platforms tested : qemu_uefi-amd64 🟢 ok bpf.ig; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.basic; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.cloudinit.basic; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.cloudinit.multipart-mime; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.cloudinit.script; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.disk.raid0.data; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.disk.raid0.root; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.disk.raid1.data; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.disk.raid1.root; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.etcd-member.discovery; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.etcd-member.etcdctlv3; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.etcd-member.v2-backup-restore; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.filesystem; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.flannel.udp; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.flannel.vxlan; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.instantiated.enable-unit; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.kargs; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.luks; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.oem.indirect; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.oem.indirect.new; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.oem.regular; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.oem.regular.new; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.oem.reuse; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.oem.wipe; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.partition_on_boot_disk; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.symlink; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.translation; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v1.btrfsroot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v1.ext4root; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v1.groups; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v1.once; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v1.sethostname; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v1.users; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v1.xfsroot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v2.btrfsroot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v2.ext4root; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v2.users; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v2.xfsroot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v2_1.ext4checkexisting; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v2_1.swap; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.ignition.v2_1.vfat; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.install.cloudinit; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.internet; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.locksmith.cluster; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.network.initramfs.second-boot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.network.iptables; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.network.listeners; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.network.nftables; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.network.wireguard; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.omaha.ping; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.osreset.ignition-rerun; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.overlay.cleanup; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.swap_activation; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.sysext.boot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.sysext.fallbackdownload # SKIP; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.tang.nonroot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.tang.root; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.toolbox.dnf-install; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.tpm.eventlog; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.tpm.nonroot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.tpm.root; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.tpm.root-cryptenroll; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.tpm.root-cryptenroll-pcr-noupdate; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.tpm.root-cryptenroll-pcr-withupdate; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.update.badverity; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.update.reboot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.users.shells; Succeeded: qemu_uefi-amd64 (1) 🟢 ok cl.verity; Succeeded: qemu_uefi-amd64 (1) 🟢 ok confext.skiprefresh; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.auth.verify; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.ignition.groups; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.ignition.once; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.ignition.resource.local; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.ignition.resource.remote; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.ignition.resource.s3.versioned; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.ignition.security.tls; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.ignition.sethostname; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.ignition.systemd.enable-service; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.locksmith.reboot; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.locksmith.tls; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.selinux.boolean; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.selinux.enforce; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.tls.fetch-urls; Succeeded: qemu_uefi-amd64 (1) 🟢 ok coreos.update.badusr; Succeeded: qemu_uefi-amd64 (1) ❌ not ok devcontainer.docker; Failed: qemu_uefi-amd64 (1, 2, 3, 4, 5) Diagnostic output for qemu_uefi-amd64, run 5Diagnostic output for qemu_uefi-amd64, run 4Diagnostic output for qemu_uefi-amd64, run 3Diagnostic output for qemu_uefi-amd64, run 2Diagnostic output for qemu_uefi-amd64, run 1❌ not ok devcontainer.systemd-nspawn; Failed: qemu_uefi-amd64 (1, 2, 3, 4, 5) Diagnostic output for qemu_uefi-amd64, run 5Diagnostic output for qemu_uefi-amd64, run 4Diagnostic output for qemu_uefi-amd64, run 3Diagnostic output for qemu_uefi-amd64, run 2Diagnostic output for qemu_uefi-amd64, run 1🟢 ok docker.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok docker.btrfs-storage; Succeeded: qemu_uefi-amd64 (1) 🟢 ok docker.containerd-restart; Succeeded: qemu_uefi-amd64 (1) 🟢 ok docker.enable-service.sysext; Succeeded: qemu_uefi-amd64 (1) 🟢 ok docker.lib-coreos-dockerd-compat; Succeeded: qemu_uefi-amd64 (1) 🟢 ok docker.network-openbsd-nc; Succeeded: qemu_uefi-amd64 (1) 🟢 ok docker.selinux; Succeeded: qemu_uefi-amd64 (1) 🟢 ok docker.userns; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.33.8.calico.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.33.8.cilium.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.33.8.flannel.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.34.4.calico.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.34.4.cilium.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.34.4.flannel.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.35.1.calico.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.35.1.cilium.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok kubeadm.v1.35.1.flannel.base; Succeeded: qemu_uefi-amd64 (1) 🟢 ok linux.nfs.v3; Succeeded: qemu_uefi-amd64 (1) 🟢 ok linux.nfs.v4; Succeeded: qemu_uefi-amd64 (1) 🟢 ok linux.ntp; Succeeded: qemu_uefi-amd64 (1) 🟢 ok misc.fips; Succeeded: qemu_uefi-amd64 (1) 🟢 ok packages; Succeeded: qemu_uefi-amd64 (1) 🟢 ok sysext.custom-docker.sysext; Succeeded: qemu_uefi-amd64 (1) ❌ not ok sysext.custom-oem; Failed: qemu_uefi-amd64 (1, 2, 3, 4, 5) Diagnostic output for qemu_uefi-amd64, run 5Diagnostic output for qemu_uefi-amd64, run 4Diagnostic output for qemu_uefi-amd64, run 3Diagnostic output for qemu_uefi-amd64, run 2Diagnostic output for qemu_uefi-amd64, run 1🟢 ok sysext.disable-containerd; Succeeded: qemu_uefi-amd64 (1) 🟢 ok sysext.disable-docker; Succeeded: qemu_uefi-amd64 (1) 🟢 ok sysext.simple; Succeeded: qemu_uefi-amd64 (1) 🟢 ok systemd.journal.remote; Succeeded: qemu_uefi-amd64 (1) 🟢 ok systemd.journal.user; Succeeded: qemu_uefi-amd64 (1) 🟢 ok systemd.sysusers.gshadow; Succeeded: qemu_uefi-amd64 (1) |
As subjects says, drop the verbose output (annoying to scroll through it, zero useful information there). Added log scavenging - basically copies portage logs and possible configure logs in case of build failure, and uploads them to bincache.
CI: https://jenkins.flatcar.org/job/container/job/sdk/50/cldsv/
Closes #3957
Closes #3783